Computer Vision And Pattern Recognition Research Articles

Conference on Computer Vision and Pattern Recognition (CVPR) 2022

The Photogrammetric RecordVolume 37, Issue 177 p. 144-145 Notes Conference on Computer Vision and Pattern Recognition (CVPR) 2022 First published: 22 March 2022 https://doi.org/10.1111/phor.5_12405Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat No abstract is available for this article. Volume37, Issue177March 2022Pages 144-145 RelatedInformation

Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Phase error analysis and compensation for motion in high-speed phase measurement profilometry

High-speed three-dimensional (3D) measurement is increasingly important in many fields. Phase measurement profilometry (PMP) based on the binary defocusing technique has been applied to the high-speed 3D measurement scene for its higher measurement resolution and precision, and breaking the speed limitations of projector. However, because the PMP needs three phase-shifting (3-PS) patterns, motion error is inevitable to measuring dynamic objects. In this research, we construct a complete high-speed 3-PS PMP system, and re-derive two clearer motion error models than those in Weise’s research [Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2007), pp. 110.1109/CVPR.2007.383291]. Then, we theoretically analyze the effects of the truncation error on the model accuracy, especially when the motion error is higher. To this end, a polynominal-based motion error model by fitting coefficient matrix of pre-simulation is proposed. Meanwhile, its corresponding error compensation method based on local domain estimation of the Nelder-Mead algorithm is developed. Finally, both simulations and quantitative and qualitative experiments verify the accuracy and effectiveness of the proposed method, as well as demonstrate the proposed method has improvements compared with the Weise’s research.

Robustify Hand Tracking by Fusing Generative and Discriminative Methods

With the development of virtual reality (VR) technology and its applications in many fields, creating simulated hands in the virtual environment is an e ective way to replace the controller as well as to enhance user experience in interactive processes. Therefore, hand tracking problem is gaining a lot of research attention, making an important contribution in recognizing hand postures as well as tracking hand motions for VR’s input or human machine interaction applications. In order to create a markerless real-time hand tracking system suitable for natural human machine interaction, we propose a new method that combines generative and discriminative methods to solve the hand tracking problem using a single RGBD camera. Our system removes the requirement of the user having to wear to color wrist band and robustifies the hand localization even in di cult tracking scenarios.
 KeywordsHand tracking, generative method, discriminative method, human performance capture
 References[1] Malik,  A.  Elhayek,  F.  Nunnari,  K.  Varanasi, Tamaddon, A. Heloir, D. Stricker, Deephps: End-to-end estimation of 3d hand pose and shape by learning from synthetic depth, CoRR abs/1808.09208, 2018.
 URL http://arxiv.org/abs/1808.09208.
 [2] Glauser,  S.  Wu,  D.  Panozzo,  O.  Hilliges, Sorkine-Hornung, Interactive hand pose estimation using a stretch-sensing soft glove, ACM Trans, Graph. 38(4) (2019) 1-15.[3] Jiang, H. Xia, C. Guo, A model-based system for real-time articulated hand tracking using a simple data glove and a depth camera, Sensors 19 (2019) 4680. https://doi.org/10.3390/s19214680.[4] Cao, G. Hidalgo, T. Simon, S. Wei, Y. Sheikh, Openpose: Realtime multi-person 2d pose estimation using part a nity fields, CoRR abs/1812.08008, 2018.[5] Tagliasacchi, M. Schroder, A. Tkach, S. Bouaziz, M. Botsch, M. Pauly, Robust articulated-icp for real-time hand tracking, Computer Graphics Forum 34, 2015.[6] Qian, X. Sun, Y. Wei, X. Tang, J. Sun, Realtime and robust hand tracking from depth, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.[7] Tomasi, Petrov, Sastry, 3d tracking = classification + interpolation, in: Proceedings Ninth IEEE International Conference on Computer Vision 2 (2003) 1441-1448.[8] Sharp, C. Keskin, D. Robertson, J. Taylor, J. Shotton, D. Kim, C. Rhemann, I. Leichter, A. Vinnikov, Y. Wei, et al., Accurate, robust, and flexible real-time hand tracking, in: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015, pp. 3633-3642.[9] Sridhar, F. Mueller, A. Oulasvirta, C. Theobalt, Fast and robust hand tracking using detection-guided optimization, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.[10] Oikonomidis, N. Kyriazis, A.A. Argyros, Tracking the articulated motion of two strongly interacting hands, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 1862-1869.[11] Melax, L. Keselman, S. Orsten, Dynamics based 3d skeletal hand tracking, CoRR abs/1705.07640, 2017.[12] Wang, S. Paris, J. Popovic, 6d hands: Markerless hand tracking for computer aided design, 2011, pp. 549-558.
 https://doi.org/10.1145/2047196.2047269.[13]Tang, T. Yu, T. Kim, Real-time articulated hand pose estimation using semi-supervised transductive regression forests, in: 2013 IEEE International Conference on Computer Vision, 2013, pp. 3224-3231.[14] Oberweger, P. Wohlhart, V. Lepetit, Generalized feedback loop for joint hand-object pose estimation, 2019, CoRR abs/1903.10883. URL http://arxiv.org/abs/1903.10883.[15] Malik, A. Elhayek, F. Nunnari, K. Varanasi, K. Tamaddon, A. Heloir,´ D. Stricker, Deephps: End-to-end estimation of 3d hand pose and shape by learning from synthetic depth, 2018, pp. 110-119. https://doi.org/10.1109/3DV.2018.00023.[16] A. Mohammed, J.L.M. Islam, A deep learning-based end-to-end composite system for hand detection and gesture recognition, Sensors 19 (2019) 5282.  https://doi.org/10.3390/s19235282.

An automated detection model of threat objects for X-ray baggage inspection based on depthwise separable convolution

X-ray baggage inspection is an essential task to detect threat objects at important controlled access places, which can guard personal safety and prevent crime. Generally, it is carried out by screeners to visually determine whether or not a bag contains threat objects. Whereas, manual detection exhibits distinct shortcomings, from high detection errors to different detection results produced by screeners. These limitations can be addressed by introducing automated detection model of threat objects for X-ray baggage inspection. However, existing automated detection methods cannot realize end-to-end detection and the detection results include only classification without location. In this paper, we propose an automated detection model of threat objects based on depthwise separable convolution. Our model is able to not only categorize the threat object but also locate it simultaneously. The network model has the advantage of high detection accuracy, fast computational speed, and a few parameters. Meanwhile, the precision of threat object regions is enhanced with the help of multi-scale prediction. A deformation layer is added in our model, which can provide invariance to affine warping. The experiments on the GDXray database (Mery et al. in J Nondestr Eval 34(4):42, 2015) demonstrate that the overall performance of our proposed model is superior to YOLOv3 (Redmon J and Farhadi A in YOLOv3: an incremental improvement, 2018) model, SSD (Liu et al. in SSD: single shot multibox detector. In: European Conference on Computer Vision (ECCV), pp. 21–37, 2016) model, and Tiny_YOLO (Redmon et al. in You only look once: unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788, 2015) model.

Depth-aware salient object segmentation


 
 
 Object segmentation is an important task which is widely employed in many computer vision applications such as object detection, tracking, recognition, and retrieval. It can be seen as a two-phase process: object detection and segmentation. Object segmentation becomes more challenging in case there is no prior knowledge about the object in the scene. In such conditions, visual attention analysis via saliency mapping may offer a mean to predict the object location by using visual contrast, local or global, to identify regions that draw strong attention in the image. However, in such situations as clutter background, highly varied object surface, or shadow, regular and salient object segmentation approaches based on a single image feature such as color or brightness have shown to be insufficient for the task. This work proposes a new salient object segmentation method which uses a depth map obtained from the input image for enhancing the accuracy of saliency mapping. A deep learning-based method is employed for depth map estimation. Our experiments showed that the proposed method outperforms other state-of-the-art object segmentation algorithms in terms of recall and precision.
 KeywordsSaliency map, Depth map, deep learning, object segmentation
 References[1] Itti, C. Koch, E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Transactions on pattern analysis and machine intelligence 20(11) (1998) 1254-1259.[2] Goferman, L. Zelnik-Manor, A. Tal, Context-aware saliency detection, IEEE transactions on pattern analysis and machine intelligence 34(10) (2012) 1915-1926.[3] Kanan, M.H. Tong, L. Zhang, G.W. Cottrell, Sun: Top-down saliency using natural statistics, Visual cognition 17(6-7) (2009) 979-1003.[4] Liu, Z. Yuan, J. Sun, J. Wang, N. Zheng, X. Tang, H.-Y. Shum, Learning to detect a salient object, IEEE Transactions on Pattern analysis and machine intelligence 33(2) (2011) 353-367.[5] Perazzi, P. Krähenbühl, Y. Pritch, A. Hornung, Saliency filters: Contrast based filtering for salient region detection, in: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, IEEE, 2012, pp. 733-740.[6] M. Cheng, N.J. Mitra, X. Huang, P.H. Torr, S.M. Hu, Global contrast based salient region detection, IEEE Transactions on Pattern Analysis and Machine Intelligence 37(3) (2015) 569-582.[7] Borji, L. Itti, State-of-the-art in visual attention modeling, IEEE transactions on pattern analysis and machine intelligence 35(1) (2013) 185-207.[8] Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: Visualising image classification models and saliency maps, arXiv preprint arXiv:1312.6034.[9] Li, Y. Yu, Visual saliency based on multiscale deep features, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 5455-5463.[10] Liu, J. Han, Dhsnet: Deep hierarchical saliency network for salient object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 678-686.[11] Achanta, S. Hemami, F. Estrada, S. Susstrunk, Frequency-tuned saliency detection model, CVPR: Proc IEEE, 2009, pp. 1597-604.Fu, J. Cheng, Z. Li, H. Lu, Saliency cuts: An automatic approach to object segmentation, in: Pattern Recognition, 2008. ICPR 2008. 19th International Conference on, IEEE, 2008, pp. 1-4Borenstein, J. Malik, Shape guided object segmentation, in: Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, Vol. 1, IEEE, 2006, pp. 969-976.Jiang, J. Wang, Z. Yuan, T. Liu, N. Zheng, S. Li, Automatic salient object segmentation based on context and shape prior., in: BMVC. 6 (2011) 9.Ciptadi, T. Hermans, J.M. Rehg, An in depth view of saliency, Georgia Institute of Technology, 2013.Desingh, K.M. Krishna, D. Rajan, C. Jawahar, Depth really matters: Improving visual salient region detection with depth., in: BMVC, 2013.Li, J. Ye, Y. Ji, H. Ling, J. Yu, Saliency detection on light field, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2806-2813.Koch, S. Ullman, Shifts in selective visual attention: towards the underlying neural circuitry, in: Matters of intelligence, Springer, 1987, pp. 115-141.Laina, C. Rupprecht, V. Belagiannis, F. Tombari, N. Navab, Deeper depth prediction with fully convolutional residual networks, in: 3D Vision (3DV), 2016 Fourth International Conference on, IEEE, 2016, pp. 239-248.Bruce, J. Tsotsos, Saliency based on information maximization, in: Advances in neural information processing systems, 2006, pp. 155-162.Ren, X. Gong, L. Yu, W. Zhou, M. Ying Yang, Exploiting global priors for rgb-d saliency detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015, pp. 25-32.Fang, J. Wang, M. Narwaria, P. Le Callet, W. Lin, Saliency detection for stereoscopic images., IEEE Trans. Image Processing 23(6) (2014) 2625-2636.Hou, L. Zhang, Saliency detection: A spectral residual approach, in: Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, IEEE, 2007, pp. 1-8.Guo, Q. Ma, L. Zhang, Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform, in: Computer vision and pattern recognition, 2008. cvpr 2008. ieee conference on, IEEE, 2008, pp. 1-8.Fang, W. Lin, B.S. Lee, C.T. Lau, Z. Chen, C.W. Lin, Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum, IEEE Transactions on Multimedia 14(1) (2012) 187-198.Lang, T.V. Nguyen, H. Katti, K. Yadati, M. Kankanhalli, S. Yan, Depth matters: Influence of depth cues on visual saliency, in: Computer vision-ECCV 2012, Springer, 2012, pp. 101-115.Zhang, G. Jiang, M. Yu, K. Chen, Stereoscopic visual attention model for 3d video, in: International Conference on Multimedia Modeling, Springer, 2010, pp. 314-324.Wang, M.P. Da Silva, P. Le Callet, V. Ricordel, Computational model of stereoscopic 3d visual saliency, IEEE Transactions on Image Processing 22(6) (2013) 2151-2165.Peng, B. Li, W. Xiong, W. Hu, R. Ji, Rgbd salient object detection: A benchmark and algorithms, in: European Conference on Computer Vision (ECCV), 2014, pp. 92-109.Wu, L. Duan, L. Kong, Rgb-d salient object detection via feature fusion and multi-scale enhancement, in: CCF Chinese Conference on Computer Vision, Springer, 2015, pp. 359-368.Xue, Y. Gu, Y. Li, J. Yang, Rgb-d saliency detection via mutual guided manifold ranking, in: Image Processing (ICIP), 2015 IEEE International Conference on, IEEE, 2015, pp. 666-670.Katz, A. Adler, Depth camera based on structured light and stereo vision, uS Patent App. 12/877,595 (Mar. 8 2012).Chatterjee, G. Molina, D. Lelescu, Systems and methods for determining depth from multiple views of a scene that include aliasing using hypothesized fusion, uS Patent App. 13/623,091 (Mar. 21 2013).Matthies, T. Kanade, R. Szeliski, Kalman filter-based algorithms for estimating depth from image sequences, International Journal of Computer Vision 3(3) (1989) 209-238.Y. Schechner, N. Kiryati, Depth from defocus vs. stereo: How different really are they?, International Journal of Computer Vision 39(2) (2000) 141-162.Delage, H. Lee, A.Y. Ng, A dynamic bayesian network model for autonomous 3d reconstruction from a single indoor image, in: Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, Vol. 2, IEEE, 2006, pp. 2418-2428.Saxena, M. Sun, A.Y. Ng, Make3d: Learning 3d scene structure from a single still image, IEEE transactions on pattern analysis and machine intelligence 31(5) (2009) 824-840.Hedau, D. Hoiem, D. Forsyth, Recovering the spatial layout of cluttered rooms, in: Computer vision, 2009 IEEE 12th international conference on, IEEE, 2009, pp. 1849-1856.Liu, S. Gould, D. Koller, Single image depth estimation from predicted semantic labels, in: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, 2010, pp. 1253-1260.Ladicky, J. Shi, M. Pollefeys, Pulling things out of perspective, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 89-96.K. Nathan Silberman, Derek Hoiem, R. Fergus, Indoor segmentation and support inference from rgbd images, in: ECCV, 2012.Liu, J. Yuen, A. Torralba, Sift flow: Dense correspondence across scenes and its applications, IEEE transactions on pattern analysis and machine intelligence 33(5) (2011) 978-994.Konrad, M. Wang, P. Ishwar, 2d-to-3d image conversion by learning depth from examples, in: Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on, IEEE, 2012, pp. 16-22.Liu, C. Shen, G. Lin, Deep convolutional neural fields for depth estimation from a single image, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 5162-5170.Wang, X. Shen, Z. Lin, S. Cohen, B. Price, A.L. Yuille, Towards unified depth and semantic prediction from a single image, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 2800-2809.Geiger, P. Lenz, C. Stiller, R. Urtasun, Vision meets robotics: The kitti dataset, International Journal of Robotics Research (IJRR).Achanta, S. Süsstrunk, Saliency detection using maximum symmetric surround, in: Image processing (ICIP), 2010 17th IEEE international conference on, IEEE, 2010, pp. 2653-2656.E. Rahtu, J. Kannala, M. Salo, J. Heikkilä, Segmenting salient objects from images and videos, in: Computer Vision-ECCV 2010, Springer, 2010, pp. 366-37.
 
 

Deep learning-based image de-raining using discrete Fourier transformation

Single image rain streak removal is a well-explored topic in the field of computer vision. The de-raining problem is modeled as an image decomposition task where a rainy image is decomposed into rain-free background image and rain streek map. Unlike most of the existing de-raining methods, this paper attempts to decompose the rainy image in the frequency domain. The idea is inspired by pseudo-periodic characteristics of the noise signal (here the rain streaks) which leave some traces in the frequency domain, and the same can be utilized to predict the noise signal. In this paper, a deep learning-based rain streak prediction model is proposed which learns in discrete Fourier transform Oppenheim and Schafer (Discrete-TimeSignal Processing, Prentice Hall, Upper Saddle River, 1989) domain. To the best of our knowledge, this is the first approach where compressed domain coefficients are directly used as input to a deep convolutional neural network. The proposed model has been tested on publicly available synthetic datasets Fu et al. (in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https://doi.org/10.1109/CVPR.2017.186 , Yang et al. (in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https://doi.org/10.1109/CVPR.2017.183 ), Yeh et al. (in: 2015 IEEE International Conference on Consumer Electronics-Taiwan, 2015. https://doi.org/10.1109/ICCE-TW.2015.7216999 ) and results are found to be comparable with the state of the art methods in the spatial domain. The presented analysis and study have an obvious indication to extend transform domain input to train the deep learning architecture especially image de-noising like problems.

Who Gets Credit for AI-Generated Art?

SummaryThe recent sale of an artificial intelligence (AI)-generated portrait for $432,000 at Christie's art auction has raised questions about how credit and responsibility should be allocated to individuals involved and how the anthropomorphic perception of the AI system contributed to the artwork's success. Here, we identify natural heterogeneity in the extent to which different people perceive AI as anthropomorphic. We find that differences in the perception of AI anthropomorphicity are associated with different allocations of responsibility to the AI system and credit to different stakeholders involved in art production. We then show that perceptions of AI anthropomorphicity can be manipulated by changing the language used to talk about AI—as a tool versus agent—with consequences for artists and AI practitioners. Our findings shed light on what is at stake when we anthropomorphize AI systems and offer an empirical lens to reason about how to allocate credit and responsibility to human stakeholders.

Automatic Identification of Individual Primates with Deep Learning Techniques.

SummaryThe difficulty of obtaining reliable individual identification of animals has limited researcher's ability to obtain quantitative data to address important ecological, behavioral, and conservation questions. Traditional marking methods placed animals at undue risk. Machine learning approaches for identifying species through analysis of animal images has been proved to be successful. But for many questions, there needs a tool to identify not only species but also individuals. Here, we introduce a system developed specifically for automated face detection and individual identification with deep learning methods using both videos and still-framed images that can be reliably used for multiple species. The system was trained and tested with a dataset containing 102,399 images of 1,040 individuals across 41 primate species whose individual identity was known and 6,562 images of 91 individuals across four carnivore species. For primates, the system correctly identified individuals 94.1% of the time and could process 31 facial images per second.

Implementation of an Anomalous Human Activity Recognition System

This paper brings to light one of the most prominent applications of human activity recognition which is the anomaly detection. Providing security to an individual is a major concern of any society today due to the constantly increasing actions causing threats, starting from deliberate violence to an injury caused by an accident. Mere installation of a traditional closed-circuit television is not enough as it requires a human being to constantly remain alert and monitor the cameras, which is quite inefficient. This calls for the need to develop an automated security system that identifies abnormal activities in real time and brings immediate help to the victims. But it is time-consuming to recognize activities from lengthy surveillance videos and hence a new implementation idea which adaptively compresses the videos before passing it through the activity recognition system has been proposed. Adaptive video compression is a unique compression technology that compresses only the least significant parts of the video, retaining the objects of interest. By combining adaptive video compression and the convolutional 3D network with contextual multiple scales based on the temporal features (Xu et al. in Learning deep representations of appearance and motion for anomalous event detection. In: Computer Vision and Pattern Recognition (CVPR), 2015), the paper aims to provide an accurate anomalous human activity recognition system that works in real time. The method is implemented on the UCF101 crime dataset with around 13 different anomalies (Learning Spatiotemporal Features with 3D Convolutional Networks. In: International Conference on Computer Vision (ICCV), IEEE, 2015). It resulted in improved recognition performance compared to state-of-the-art techniques.

Classifying sensitive content in online advertisements with deep learning

In online advertising, an important quality control step is to audit advertising images (“creatives”) before they appear on publishers’ Web pages. This ensures that advertisements only appear on Web pages where the ad is appropriate. If a creative with sensitive content such as gambling and pornography is displayed on the wrong Web page, it can ruin the user’s experience, the publisher’s reputation, and may have legal implications. To protect against this, humans must audit every creative before it is displayed through our ad exchange; this process is costly and time-consuming. To detect sensitive content, we use a pre-trained deep convolutional neural network (Xception Chollet in: The IEEE conference on computer vision and pattern recognition (CVPR), 2017) to process the creative image, and merge its features with the historical distribution of categories associated with the creative’s landing page (the Web page that loads when the ad is clicked, which may also contain sensitive content). This representation is then passed through a series of fully connected layers to predict the sensitive category. The trained model achieves slightly better than human performance (model accuracy 99.92%; human accuracy 99.88%) on a large fraction of creatives (61%), while making 3.5 times fewer mistakes in very sensitive categories. The main challenges we faced were to detect, with high accuracy, creatives from 10 “very sensitive” categories as determined by our Creative Audit team, along with a highly imbalanced data set with 95% of creatives having no sensitive categories. This paper extends the work we described in Austin et al. (in: Proceedings of the 2018 IEEE international conference on data science and advanced analytics (DSAA), DSAA’18, 2018). It demonstrates the successful usage of deep learning in production as a method for detecting sensitive creatives, while respecting the constraints set by business.

Foreword to the Special Issue on Computer Vision-Based Approaches for Earth Observation

The five papers in this special section focus on computer vision-based approaches for Earth observation. These papers followed a series of events promoting works at the interface between computer vision and remote sensing: the special sessions organized at the Living Planet Symposium1 and the Computer Vision and Pattern Recognition (CVPR) conference (the EarthVision2 and Computer Vision for Global Challenges3 workshops). These sessions aimed at fostering collaboration between the computer vision and earth observation communities to boost automated interpretation of remotely sensed data. They also aimed at raising awareness inside the computer vision community for this highly challenging and quickly evolving field of research with a big impact on human society, economy, industry, and the planet. Submissions were invited from all areas of computer vision and image analysis relevant for, or applied to environmental remote sensing and were not limited to the papers presented at the events above. The papers retained in this special issue reflect the high variety of automatic image analysis in remote sensing.

A novel generative adversarial net for calligraphic tablet images denoising

Chinese calligraphic images have important historical and artistic value, but natural weathering and man-made decay severely damage these works, thus image denoising is an important topic to be addressed. Traditional denoising methods still leave room for improvement. In this paper, image denoising is modeled as generation of clean image by using GAN (Goodfellow I et al. Advances in Neural Information Processing Systems 2672–2680, 2014) with an embedment of residual dense blocks (Zhang Y et al. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018) that was formerly used for super resolution reconstruction. Meanwhile, a new type of noise is defined to simulate the real noise, and is used for compensation of unpaired data in the training set for GAN. The new structure, used with some preprocessing and training methods, yield satisfactory results compared to known denoising methods.

Image-Based Geo-Localization Using Satellite Imagery

The problem of localization on a geo-referenced satellite map given a query ground view image is useful yet remains challenging due to the drastic change in viewpoint. To this end, in this paper we work on the extension of our earlier work on the Cross-View Matching Network (CVM-Net) (Hu et al. in IEEE conference on computer vision and pattern recognition (CVPR), 2018) for the ground-to-aerial image matching task since the traditional image descriptors fail due to the drastic viewpoint change. In particular, we show more extensive experimental results and analyses of the network architecture on our CVM-Net. Furthermore, we propose a Markov localization framework that enforces the temporal consistency between image frames to enhance the geo-localization results in the case where a video stream of ground view images is available. Experimental results show that our proposed Markov localization framework can continuously localize the vehicle within a small error on our Singapore dataset.

Computer Analysis of Architecture Using Automatic Image Understanding

In the past few years, computer vision and pattern recognition systems have been becoming increasingly more powerful, expanding the range of automatic tasks enabled by machine vision. Here we show that computer analysis of building images can perform quantitative analysis of architecture, and quantify similarities between city architectural styles in a quantitative fashion. Images of buildings from 18 cities and three countries were acquired using Google StreetView, and were used to train a machine vision system to automatically identify the location of the imaged building based on the image visual content. Experimental results show that the automatic computer analysis can automatically identify the geographical location of the StreetView image. More importantly, the algorithm was able to group the cities and countries and provide a phylogeny of the similarities between architectural styles as captured by StreetView images. These results demonstrate that computer vision and pattern recognition algorithms can perform the complex cognitive task of analyzing images of buildings, and can be used to measure and quantify visual similarities and differences between different styles of architectures. This experiment provides a new paradigm for studying architecture, based on a quantitative approach that can enhance the traditional manual observation and analysis. The source code used for the analysis is open and publicly available.

Learning 3D Shape Completion Under Weak Supervision

We address the problem of 3D shape completion from sparse and noisy point clouds, a fundamental problem in computer vision and robotics. Recent approaches are either data-driven or learning-based: Data-driven approaches rely on a shape model whose parameters are optimized to fit the observations; Learning-based approaches, in contrast, avoid the expensive optimization step by learning to directly predict complete shapes from incomplete observations in a fully-supervised setting. However, full supervision is often not available in practice. In this work, we propose a weakly-supervised learning-based approach to 3D shape completion which neither requires slow optimization nor direct supervision. While we also learn a shape prior on synthetic data, we amortize, i.e., learn, maximum likelihood fitting using deep neural networks resulting in efficient shape completion without sacrificing accuracy. On synthetic benchmarks based on ShapeNet (Chang et al. Shapenet: an information-rich 3d model repository, 2015. arXiv:1512.03012) and ModelNet (Wu et al., in: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), 2015) as well as on real robotics data from KITTI (Geiger et al., in: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), 2012) and Kinect (Yang et al., 3d object dense reconstruction from a single depth view, 2018. arXiv:1802.00411), we demonstrate that the proposed amortized maximum likelihood approach is able to compete with the fully supervised baseline of Dai et al. (in: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR), 2017) and outperforms the data-driven approach of Engelmann et al. (in: Proceedings of the German conference on pattern recognition (GCPR), 2016), while requiring less supervision and being significantly faster.

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

The problem of visual question answering (VQA) is of significant importance both as a challenging research question and for the rich set of applications it enables. In this context, however, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in VQA models that ignore visual information, leading to an inflated sense of their capability. We propose to counter these language priors for the task of VQA and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset (Antol et al., in: ICCV, 2015) by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at http://visualqa.org/ as part of the 2nd iteration of the VQA Dataset and Challenge (VQA v2.0). We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. We also present interesting insights from analysis of the participant entries in VQA Challenge 2017, organized by us on the proposed VQA v2.0 dataset. The results of the challenge were announced in the 2nd VQA Challenge Workshop at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017. Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.

Sparse representation based image deblurring model under random-valued impulse noise

In this article, we introduce a new patch-based model for restoring images simultaneously corrupted by blur and random-valued impulse noise. The model involves a $$l_0$$ -norm data-fidelity term, a sparse representation prior over learned dictionaries, and the total variation (TV) regularization. Unlike previous works Cai et al. (Inverse Probl Imaging 2(2):187–204, 2008), Ma et al. (SIAM J Imaging Sci 6(4):2258–2284, 2013), one-phase approach is utilized for random-valued impulse noise. As in Yuan and Ghanem (IEEE conference on computer vision and pattern recognition (CVPR), pp 5369–5377, 2015), the $$l_0$$ data-fitting term plays an influential role for removing random-valued impulse noise. Moreover, the sparse representation prior enables to preserve textures and details efficiently, whereas TV regularization locally smoothes images while keeping sharp edges. To handle nonconvex and nondifferentiable terms, we adopt a variable splitting scheme, and then the penalty method and alternating minimization algorithm are employed. This results in an efficient iterative algorithm for solving our model. Numerical results are reported to show the effectiveness of the proposed model compared with the state-of-the-art methods.

PMHI: Proposals From Motion History Images for Temporal Segmentation of Long Uncut Videos

This letter proposes a method for the generation of temporal action proposals for the segmentation of long uncut video sequences. The presence of consecutive multiple actions in video sequences makes the temporal segmentation a challenging problem due to the unconstrained nature of actions in space and time. To address this issue, we exploit the nonaction segments present between the actual human actions in uncut videos. From the long uncut video, we compute the energy of consecutive nonoverlapping motion history images (MHIs), which provides spatiotemporal information of motion. Our proposals from MHIs (PMHI) are based on clustering the MHIs into actions and nonaction segments by detecting minima from the energy of MHIs. PMHI efficiently segments the long uncut videos into a small number of nonoverlapping temporal action proposals. The strength of PMHI is that it is unsupervised, which alleviates the requirement for any training data. Our temporal action proposal method outperforms the existing proposal methods on the Multi-view Human Action video (MuHAVi)-uncut and Computer Vision and Pattern recognition (CVPR) 2012 Change Detection datasets with an average recall rate of 86.1% and 86.0%, respectively.

Foreword to the Special Issue on “GeoVision: Computer Vision for Geospatial Applications”

The nine papers in this special section focus on the development of new computer vision techniques for the interpretation of remote sensing images. These papers represent a follow-up of two workshops held in conjunction with the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015, that was held in Boston, MA, EARTHVISION 2015 and MSF 2015. The purpose of both workshops and of this special issue is to foster fruitful collaboration of computer vision, Earth observation, and geospatial analysis communities.